Leader election with lifetime term

ABSTRACT

A processing device receives request from a process of a plurality of processes of a clusterized service, to attempt to obtain exclusive access to a predetermined resource associated with a leader state of the plurality of processes. Responsive to successfully obtaining the exclusive access to the predetermined resource, the processing device enables the process to enter the leader state. The processing device enables the process to stay in the leader state for the lifetime of the process.

TECHNICAL FIELD

Embodiments of the present invention relate generally to computersystems, and more specifically to methods of performing leader electionwith lifetime term.

BACKGROUND

Data centers may include clusters consisting of multiple hosts (e.g.,physical servers). Each of the hosts may run one or more containers,such that each container provides an isolated execution environment inthe user space of the host operating system, sharing the kernel withother containers. Each container executes one or more related processesproviding a certain service (e.g., an HTTP server, a database server,etc.).

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, and can be more fully understood with reference to thefollowing detailed description when considered in connection with thefigures in which:

FIG. 1 is a block diagram of a computer system representing a hostsystem, in accordance with one implementation.

FIG. 2 depicts a block diagram of an example computer system inaccordance with one or more aspects of the present disclosure;

FIG. 3 is a flow diagram illustrating one embodiment for a method ofperforming leader election with lifetime term in a Kubernetesenvironment.

FIG. 4 is a flow diagram illustrating another embodiment for a method ofprocessing a leader election module with lifetime term.

FIG. 5 is a block diagram of an example computer system, in accordancewith one implementation.

FIG. 6 is a flow diagram illustrating a method for monitoring thelifetime of a process in the leader state, according to one embodiment.

FIG. 7 illustrates a diagrammatic representation of a machine in theexemplary form of a computer system.

DETAILED DESCRIPTION

Described herein are methods and systems for leader election with alifetime term. In a clusterized service, which may include a pluralityof instances of the same executable code (referred to as “plurality ofprocesses herein”), the leader process may need to be elected insituations when only one process is expected to provide authoritativeresponse to incoming client requests and/or to perform load distributionamong other processes of the clusterized service. A cluster is a set ofloosely or tightly connected computers that work together so that theymay be viewed, in many respects, as single system consisting of multiplenodes. For example, a virtual router service may include one master(“leader”) instance and one or more standby instances, one of whichwould become the master instance if the current master instance becomesunresponsive or is terminated. As an example of the leader process, aweb service may need to become the leader process in order to access adatabase table that enforces uniqueness constraints to insert newrecords. In another example, the leader process may be a thread in amulti-threaded application attempting to update a global object residingin a memory (e.g., the main memory or a storage memory).

In common implementations, the leader process is expected to continuallysend heartbeat signals to confirm that it is still alive and responsive,while the other processes of the cluster are expected to perform newleader election responsive to failing to receive the heartbeat signalwithin a predetermined period of time. In these implementations, anexisting leader process may, upon expiration of a timeout or occurrenceof another triggering event, either renew its lease to stay in theleader role or step down so another process may be elected to the leaderrole. If the lease is not renewed or a leader has stepped down, a newleader election is performed. In these implementations, it would bepossible for two leaders to coexist concurrently at least for a briefperiod of time, potentially causing memory corruption or data collision.In one example, for the lease management solution to be effective atpreventing dual leadership, the clocks of all candidate processes shouldbe synchronized so that lease expiration would be detected by eachprocess at the same time. For example, the concurrent leader processesmay coexist because the clock of one process may not be synchronizedwith the clock of the other process, thus the indication that the leasehas not been renewed may be perceived by each process at a differenttime. Consequently, this variance in detection of lease renewal mayresult in unpredictable results because two leader processes are presentand may attempt to update the same data currently, for example. Thissynchronization and delay management aspects of the traditionalsolutions add overhead and complexities in the hardware and softwarelayers of a leader election system.

Aspects of the present disclosure address the above-noted and otherdeficiencies by introducing the lifetime term for the leader process,thus eliminating the need for sending heartbeats or otherwise renewingthe leader status. The leader election procedure may involve obtainingexclusive access to a predetermined resource: one or more candidateprocesses of the plurality of processes of the cluster would attempt toobtain exclusive access to the predetermined resource, and only onecandidate process would succeed (by virtue of the access beingexclusive), thus becoming the leader for its lifetime. In anillustrative example, the predetermined resource may be represented by amemory object (e.g., a lock or a ConfigMap object in a Kubernetesenvironment) or a file with a predetermined name. When a process obtainsexclusive access to the predetermined resource, such a process becomesthe leader process, while the other processes in the cluster maydetermine that a leader process exists because attempts to gainexclusive access to the predetermined resource by the other processeswould fail. When the leader process is eventually terminated, theexclusive access to the predetermined resource is released by anoperating system process and another process may succeed in obtaining anexclusive access to the predetermined resource and may become the newleader with a lifetime term.

In an illustrative example, multiple processes within a cluster maycompete to enter a leader state. A cluster is a set of loosely ortightly connected computers that work together so that they may beviewed, in many respects, as single system consisting of multiple nodes.The leader state is an exclusive state which at any given moment of timeis associated with not more than one process of the cluster. As notedherein above, to enter the leader state, a process may attempt to obtainexclusive access to a predetermined resource which is employed as themarker for the leader state of the cluster. In one implementation, in aKubernetes environment, the predetermined resource may be a representedby an object of a predetermined type (e.g., ConfigMap object) withrespect to which the environment would enforce uniqueness constraints.Thus, in order to enter the leader state, a process would attempt tocreate the ConfigMap. Kubernetes is an open-source system fordeployment, scaling, and management of containerized application. AConfigMap is an object that binds configuration information tocontainers and system components at runtime to enable sharing ofinformation among system components, including processes. When a processis able to create the ConfigMap object, the process enters the leaderstate and further attempts to create the ConfigMap by other processeswould fail. The ConfigMap object may have a unique name known to themultiple processes in the cluster to ensure exclusive access to theobject as follows: when the leader process creates the ConfigMap, theoperating system may prevent subsequent attempts to create a ConfigMapwith the known name because one already exists. Further, the ConfigMapmay have a property identifying its owner (i.e., the owner reference).The owner reference of the ConfigMap identifies the leader process, suchthat when the leader process is terminated the ConfigMap may be deletedfrom memory, for example by a garbage collector process.

In another illustrative example, the predetermined resource may berepresented by a file having a predetermined name known to the processesin the cluster as identifying the leader state marker file. In order tofor a process to enter a leader state, the process may attempt to createthe file at a predetermined location in the file system. Further, theoperating system may ensure exclusive access to the file by causingsubsequent attempts to create the file with the predetermined name inthe predetermined location to fail if the file already exists. Theoperating system process may also ensure the lifetime term for theleader process by maintaining control over the file at the predeterminedlocation so that it may not be deleted by another process. When theprocess in the leader state is eventually terminated, an operatingsystem process may delete the file, thus signaling to the otherprocesses that no leader process currently exists in the cluster. Theoperating system process may be represented by any kernel process thatis responsible for cleaning up data items and files that are not in use.

In an embodiment of the present disclosure, when a process enters theleader state, by successfully gaining exclusive access to thepredetermined resource, the process is expected to stay in the leaderstate for the lifetime of the process. Thus, the process would not needto renew the leader state lease. Because the process is expected tomaintain the leader state for its lifetime, the process also wouldmaintain the exclusive access to the predetermined resource, thussignaling to the other processes of the cluster that the leader statemay be occupied. Accordingly, when another processes attempt to obtainexclusive access the predetermined resource, such an attempt would failbecause the exclusive access to predetermined resource is alreadyobtained by the leader process. The processes competing for the leaderstate may periodically repeat the attempt to obtain exclusive access tothe predetermined resource. In one example, processes may repeat theattempt to obtain exclusive access to the predetermined resource at theexpiration of a configurable interval of time. The configurable intervalof time may be defined as a number of seconds. After a failed attempt toobtain exclusive access to the predetermined resource, a process mayreset a counter to zero and then may start to increment the counterevery second. When the value of the counter reaches the configurableinterval of time, the process may repeat the attempt to obtain exclusiveaccess to the predetermined resource and may reset the counter to zero.The process of resetting the counter and repeating the attempt to obtainexclusive access to the predetermined resource may continue until theprocess is eventually able to gain the exclusive access to thepredetermined resource and enter the leader state. The predeterminedresource may have an owner reference parameter identifying the leaderprocess.

In one embodiment, when the lifetime of the process in the leader stateeventually ends because the process has been terminated, an operatingsystem process (such as a garbage collector process) may detect that theleader process has been terminated, and thus the predetermined resourceutilized as the leader marker should be released. The operating systemprocess may make that determination because the owner reference of thepredetermined resource is associated with the leader process that is nowterminated. The operating system process may then release the exclusiveaccess to the predetermined resource, thus signaling to the otherprocesses in the environment that the leader state may be available. Inone example, the operating system process may be a garbage collectorprocess and the predetermined resource may be a memory object. Inanother example, another operating system process may delete a fileassociated with the leader state when detecting that the leader processhas been terminated.

This method of leader election with lifetime term enables a reliableprocess of electing a leader without the need for lease renewal orhandling of leader stepdown decisions, resulting in a simplified processof leader election with minimal dependency on synchronized hardware andsoftware resources. Further, the method disclosed herein eliminates thepossibility of having two concurrent leaders because a process in theleader state may continue to be the leader, with no additional leaderelection tasks, until the end of its lifetime.

In the following description, numerous details are set forth. It will beapparent, however, to one skilled in the art, that the present inventionmay be practiced without these specific details. In some instances,well-known structures and devices are shown in block diagram form,rather than in detail, in order to avoid obscuring embodiments of thepresent invention.

Various aspects of the above referenced methods and systems aredescribed in details herein below are targeted at improving theefficiency and reliability of resource contention handling indistributed computing environments by enabling a leader election processwith lifetime term.

FIG. 1 is a block diagram that illustrates an embodiment of a computersystem (referred to herein as node 100) that hosts one or morecontainers 125A-C in a Kubernetes environment. Node 100 may comprise oneor more computing devices with one or more processors communicativelycoupled to memory devices and input/output (I/O) devices. Node 100 mayrefer to a computing device (e.g., physical machine), a virtual machine,or a combination thereof. A node may provide one or more levels ofvirtualization such as hardware level virtualization, operating systemlevel virtualization, other virtualization, or a combination thereof.The hardware level virtualization may involve a hypervisor (e.g.,virtual machine monitor) that emulates portions of a physical system andmanages one or more virtual machines. In contrast, operating systemlevel virtualization may include a single operating system kernel thatmanages multiple isolated virtual containers. Each virtual container mayshare the kernel of the underlying operating system without requiringits own kernel. The memory 128 may include volatile memory devices(e.g., random access memory (RAM)), non-volatile memory devices (e.g.,flash memory), and/or other types of memory devices.

Node 100 may provide operating system level virtualization by running acomputer program that provides computing resources to one or morecontainers 125A-C. Operating system level virtualization may beimplemented within the kernel of operating system 123 and may enable theexistence of multiple isolated containers. In one example, operatingsystem level virtualization may not require hardware support and mayimpose little to no overhead because programs within each of thecontainers may use the system calls of the same underlying operatingsystem 123. Operating system level virtualization may provide resourcemanagement features that isolate or limit the impact of one container(e.g., container 125A) on the resources of another container (e.g.,container 125B or 125C).

The operating system level virtualization may provide a pool ofcomputing resources that are accessible by container 125A and areisolated from one or more other containers (e.g., container 125B). Thepool of resources may include filesystem resources (e.g., particularvolumes), network resources (e.g., particular network interfaces,sockets, addresses, or ports), memory resources (e.g., particular memoryportions), other computing resources, or a combination thereof. Theoperating system level virtualization may also limit (e.g., isolate) acontainer's access to one or more computing resources by monitoring thecontainers activity and restricting the activity in view of one or morelimits. The limits may restrict the rate of the activity, the aggregateamount of the activity, or a combination thereof. The limits may includeone or more of filesystem limits, disk limits, input/out (I/O) limits,memory limits, CPU limits, network limits, other limits, or acombination thereof.

Operating system 123 may provide containers 125A-C with access tocomputing resources. The operating system may wrap one or more processes(e.g., an instance of a particular service) in a complete filesystemthat contains the code, runtime, system tools, system libraries, andother data present on the node that may be used by the processesexecuting within the container. Operating system 123 may manage theexecution of processes 128A-C.

Each of the containers 125A-C may refer to a resource-constrainedprocess space of node 100 that may execute functionality of a program.Containers 125A-C may be referred to as a user-space instances or avirtualization engines (VE) and may appear to a user as a standaloneinstance of the user space of operating system 123. Each of thecontainers 125A-C may share the same kernel but may be constrained touse only a defined set of computing resources (e.g., CPU, memory, I/O).An embodiment of the disclosure may create one or more containers tohost a framework or provide other functionality of a service (e.g., webapplication functionality, database functionality) and may therefore bereferred to as “service containers” or “application containers.”

Pods 129A and 129B may be data structures that are used to organize oneor more containers 125A-C and enhance sharing between containers, whichmay reduce the level of isolation between containers within the samepod. Each pod may include one or more containers that share somecomputing resources with another container associated with the pod. Eachpod may be associated with a unique identifier, which may be anetworking address (e.g., IP address), that allows applications to useports without a risk of conflict. A pod may be associated with a pool ofresources and may define a volume, such as a local disk directory or anetwork disk and may expose the volume to one or more (e.g., all) of thecontainers within the pod. In one example, all of the containersassociated with a particular pod may be co-located on the same node 100.In another example, the containers associated with a particular pod maybe located on different nodes that are on the same or different physicalmachines.

Processes 128A-C may be instances of any computing services in aclusterized environment that can receive, transmit, or process requestsand perform one or more computing tasks in view of the requests.Further, processes 128A-C may be a plurality of instances of the sameexecutable code (referred to as “plurality of processes herein”). In oneexample, processes 128A-C may include one or more web services, databaseservices, filesystem services, networking services, messaging services,load balancing services, clustering services, configuration managementservices, other services, or a combination thereof. Each process may beexecuted on node 100 and may function as a client process, a serverprocess, or a combination thereof.

Data store 132 may be represented by any repository that is capable ofstoring one or more files, storage metadata, storage lease information,disk images, or a combination thereof. Data store 132 may be accessibleto node 100 via network 140. Data store 132 may include block-basedstorage devices, file-based storage devices, other storage device, or acombination thereof. Block-based storage devices may provide access toconsolidated block-based (e.g., block-level) data storage and mayinclude one or more devices (e.g., Storage Area Network (SAN) devices).Block-based storage devices may be accessible over network 140 and mayappear to an operating system of a computing device as locally attachedstorage. File-based storage devices may provide access to consolidatedfile-based (e.g., file-level) data storage using one or more datastorage devices (e.g., Network Attached Storage (NAS) devices) that maybe accessible over network 140. In one example, a secondary storage withdata store 132 may employ file-based storage that may provide access toa locked resource 134A that may need to be accessed by the leaderprocess.

Locked resource 134A may be a software or hardware resource that may beaccessed by one process at a time via network 140, thus in order toaccess locked resource 134A a process needs to be the leader process. Inone example, locked resource 134A may be a database table withuniqueness constraint where records may be created or updated by oneprocess at a time because one or more columns in the table requireunique values. In another example, the locked resource may be a globalobject residing in a memory (e.g., the main memory or a storage memory)and the leader process may be a thread in a multi-threaded applicationattempting to update the global object. Similarly, locked resource 134Bmay be a local software or hardware resource that is hosted locally onnode 100, and that may be accessed by one process at a time. The leaderprocess may gain access to locked resources 134A-B through the creationof ConfigMap object 124.

ConfigMap 124 may be an object that binds configuration information tocontainers and system components at runtime to enable sharing ofinformation among processes 128A-C. When one of processes 128A-C createsConfigMap 124, the process enters a leader state and further attempts tocreate ConfigMap 124 by the other processes would fail. ConfigMap 124may have a unique name known to processes 128A-C. Further, ConfigMap 124may have an owner reference that is associated with the process in theleader state. The owner reference of the ConfigMap ties the ConfigMap tothe process in the leader state, such that when the leader process isterminated, memory management process 126 may delete ConfigMap 124 frommemory.

Memory management process 126 may be responsible for monitoring memoryallocation and reclaiming memory occupied by objects that are no longerin use by any process. When the lifetime of the process in the leaderstate eventually ends, the leader state may become available to otherprocesses. This is accomplished via memory management process 126 bydetecting that ConfigMap 124 should be deleted from memory when theleader process is no longer alive. Memory management process 126 mayestablish that ConfigMap 124 should be deleted because the ownerreference of ConfigMap 124 is associated with a process that has beenterminated. Memory management process 126 may then delete ConfigMap 124from memory, thus signaling to the other processes that the leader statemay be available. In one example, memory management process 126 may be agarbage collector process that identifies an object for deletion basedon the allocation status of its parent process.

FIG. 2 depicts a block diagram of an example computer system 200 inaccordance with one or more aspects of the present disclosure. Computersystem 200 may include one or more processing devices and one or morememory devices. In the example shown, computer system 200 may include aleader state entering software module 210, a leader lifetime monitoringsoftware module 220, and a timeout expiration software module 230. Theleader state entering module 210 may be where processes attempt to entera leader state to perform a task that may be performed by one process ata time. In one example, the processes may need to enter the leader stateto execute a workload that may involve creating records in a data storethat enforces uniqueness constraints. To enter the leader state, aprocess may attempt to obtain exclusive access to a predeterminedresource which is employed as the marker for the leader state of thecluster. In one example, the predetermined resource may be a file with apredetermined name known to processes in the cluster as identifying theleader state marker file. Obtaining an exclusive access to thepredetermined resources may involve creating the file in a local orremote file system. When the file is successfully created by a process,the process enters the leader state. Further, an operating systemprocess may ensure exclusive access to the file by causing subsequentattempts to create the file with the predetermined name to fail if thefile already exists. When a process enters the leader state, bysuccessfully creating the file, the process may stay in the leader statefor its lifetime.

Leader lifetime monitoring module 220 may be responsible for cleaning upresources when the leader process is terminated. The process in theleader state may maintain exclusive access to a predetermined resourcethrough its lifetime, thus signaling to the other processes that theleader state may be occupied. When the process in the leader state iseventually terminated, an operating system process may subsequentlyrelease the exclusive access to the predetermined resource, thussignaling to the other processes that the leader state may be available.For example, the operating system process may be a garbage collectorprocess or another kernel process that is responsible for data clean up.In an illustrative example, the predetermined resource may be a filewith a predetermined name and releasing the exclusive access to thepredetermined resource may be deleting the physical file from a filesystem. In this example, when the lifetime of the process in the leaderstate eventually ends, the operating system process may detect that theleader process has been terminated and a file associated with the leaderstate may be deleted from the file system. In doing so, the operatingsystem process signals the other processes in the cluster that theleader state may be available and subsequent attempts to create the fileand enter the leader state by a second process would succeed. The secondprocess would then stay in the leader state for its lifetime.

The timeout expiration module 230 may be responsible for determining thetime period that a second process may need to wait before repeating itsattempt to enter the leader state. When a first process enters theleader state, by successfully gaining an exclusive access to apredetermined resource, the first process may stay in the leader statefor its lifetime. Accordingly, when a second process attempts to gainexclusive access to the predetermined resource, the attempt would failbecause the process in the leader state already holds the exclusiveaccess to the resource. The second process competing for the leaderstate may repeat the attempt to obtain exclusive access to thepredetermined resource periodically. In one example, the second processmay repeat the attempt to gain an exclusive access to the predeterminedresource at the expiration of a configurable interval of time. Theconfigurable interval of time, referenced herein as the timeoutexpiration period, may be defined as a number of seconds. After a failedattempt to access the resource exclusively, the second process may reseta counter to zero and then may start to increment the counter everysecond. When the value of the counter reaches the configurable intervalof time, the second process may repeat the attempt to access theresource and reset the counter to zero. The sequence of resetting thecounter and repeating the attempt may continue until the second processis eventually able to gain the exclusive access to the predeterminedresource and enter the leader state.

In another example, the timeout expiration period may be determinedbased on an exponential backoff algorithm where the timeout period maybe longer with each repetition. Exponential backoff refers to analgorithm used to space out repeated transmission of the same block ofdata, often to avoid network congestion. The transmission may be delayedby an amount of time derived from the number of attempts to retransmit,according to the formula: time delay=2{circumflex over ( )}R−1, where Ris the repetition count. For example, after the first failed attempt,the process may wait for (2{circumflex over ( )}1−1)=1 second, and afterthe second attempt, the process may wait for (2{circumflex over( )}2−1)=3 seconds, and so on. As the number of attempts increases, thetimeout period for delay may increase exponentially. Further, to avoidunreasonably long delays with repeated attempts, or when there are alarge number of processes competing to enter the leader state, a ceilingvalue for the timeout period may be defined so that after a certainnumber of repetitions, the timeout value may reach the ceiling value andmay not increase any further.

For simplicity of explanation, the methods of this disclosure aredepicted and described as a series of acts. However, acts in accordancewith this disclosure can occur in various orders and/or concurrently,and with other acts not presented and described herein. Furthermore, notall illustrated acts may be required to implement the methods inaccordance with the disclosed subject matter. In addition, those skilledin the art will understand and appreciate that the methods couldalternatively be represented as a series of interrelated states via astate diagram or events. Additionally, it should be appreciated that themethods disclosed in this specification are capable of being stored onan article of manufacture to facilitate transporting and transferringsuch methods to computing devices. The term article of manufacture, asused herein, is intended to encompass a computer program accessible fromany computer-readable device or storage media. Each method describedherein and/or each of its individual functions, routines, subroutines,or operations may be performed by one or more processing devices of thecomputer system (e.g., node 100 of FIG. 1) implementing the method. Incertain implementations, the method may be performed by a singleprocessing thread. Alternatively, the method may be performed by two ormore processing threads, each thread executing one or more individualfunctions, routines, subroutines, or operations of the method. In anillustrative example, the processing threads implementing the method maybe synchronized (e.g., using semaphores, critical sections, and/or otherthread synchronization mechanisms). Alternatively, the processingthreads implementing the method may be executed asynchronously withrespect to each other.

FIG. 3 is flow diagrams of an embodiment of a method related to leaderelection with lifetime term in a Kubernetes environment. The method isperformed by processing logic that may include hardware (circuitry,dedicated logic, etc.), software (such as is run on a general purposecomputer system or a dedicated machine), or a combination of both. Inone implementation, the method may be performed by one or more leaderelection modules from one or more computer systems, referred to hereinas processing logic. The flow diagram illustrates one embodiment for amethod 300 of processing leader election module. At block 302 of method300, processing logic may attempt to create a ConfigMap objectassociated with a leader state of the processes in the cluster. AConfigMap is an object that binds configuration information tocontainers and system components at runtime to enable sharing ofinformation among system components. In one example, processes may needto enter the leader state to execute a workload that may involvecreating records in a data store that enforces uniqueness constraints.To enter the leader state, a process may attempt to obtain exclusiveaccess to a predetermined resource which is employed as the marker forthe leader state of the cluster. In one example, the predeterminedresource may be a ConfigMap and obtaining an exclusive access to thepredetermined resources may be creating the ConfigMap. When a processcreates the ConfigMap, the process enters the leader state, furtherattempts to create the ConfigMap by the other processes would fail. TheConfigMap may have a unique name known to the processes in the clusterand may also have an owner reference that is associated with the processin the leader state, tying the ConfigMap to the process in the leaderstate.

At block 317, the processing logic determines whether the ConfigMap hasbeen created successfully. If ConfigMap has been created by theprocessing logic, then the leader state may have been vacant and theprocess that created the ConfigMap may be the new leader, thus theprocess may enter the leader state at block 330. Other processes may notbe allowed to create the ConfigMap for the lifetime of the process. Theprocess stays in the leader state for its lifetime at 340. Further, theConfigMap may be associated with an owner reference parameter equal tothe identifier of the process currently in the leader state.

If the processing logic determines that the ConfigMap has not beensuccessfully created, indicating that another process may be in theleader state at 325, the processing logic may wait until the expirationof a timeout period at 326 before repeating the attempt to create theConfigMap. In one example, the processing logic may repeat the attemptto create the ConfigMap object at the expiration of a configurableinterval of time. The configurable interval of time, referenced hereinas the timeout expiration period, may be defined as a number of seconds.After a failed attempt to create ConfigMap, a process may reset acounter to zero and then may start to increment the counter everysecond. When the value of the counter reaches the configurable intervalof time, the processing logic may repeat the attempt to create theConfigMap and reset the counter to zero. The process of resetting thecounter and repeating the attempt to create the ConfigMap may continueuntil the processing logic is eventually able to create the ConfigMapand enter the leader state.

In another example, the timeout expiration period may be determinedbased on an exponential backoff algorithm where the timeout period maybe longer with each repetition. Exponential backoff may refer to analgorithm used to space out repeated transmissions of the same block ofdata, often to avoid network congestion. The transmission may be delayedby an amount of time derived from the number of attempts to retransmit,according to the formula: time delay=2{circumflex over ( )}R−1, where Ris the repetition count. In one example, in order to avoid unreasonablylong delays with repeated attempts or when there are a large number ofprocesses competing to enter the leader state, a ceiling value for thetimeout period may be defined so that after a certain number ofrepetitions, the timeout value reaches the ceiling value and does notincrease any further.

After waiting for the expiration of the timeout period, the processinglogic may repeat the attempt to create the ConfigMap to enter the leaderstate at 302.

FIG. 4 is a flow diagram illustrating one embodiment for a method 400 ofprocessing a leader election module with lifetime term. At block 402 ofmethod 400, processing logic may attempt to obtain exclusive access to apredetermined resource associated with a leader state of a plurality ofprocesses. In one example, the predetermined resource may be a ConfigMapassociated with a leader state of the plurality of processes. Processesmay need to enter the leader state to execute a workload that mayinvolve creating records in a data store that enforces uniquenessconstraints. Obtaining an exclusive access to the predetermined resourcemay be creating the ConfigMap. When the ConfigMap is created by aprocess, the process enters the leader state. Further attempts to createthe ConfigMap by the other processes of the plurality of processes wouldfail.

At block 405, responsive to successfully obtaining the exclusive accessto the predetermined resource, the processing logic may enable a processto enter the leader state. Other processes may not be allowed to createthe ConfigMap for the lifetime of the process currently in the leaderstate. The ConfigMap may be identified by a unique name known to theprocesses. The ConfigMap may have an owner reference identifying theprocess currently in the leader state. Finally, at 418, the processinglogic may maintain the process in the leader state for the lifetime ofthe process. Further, the process in the leader state may maintainexclusive access of the ConfigMap for the duration of the process'slifetime, thus signaling to the other processes that the leader statemay be occupied.

FIG. 5 is a block diagram of an example computer system, in accordancewith one implementation. Computer system 550 may comprise one or morecomputing devices with one or more processors communicatively coupled tomemory devices and input/output (I/O) devices. Computer system 550 mayrefer to a computing device (e.g., physical machine), a virtual machine,or a combination thereof. Computer system 550 may provide operatingsystem level virtualization by running a computer program that providescomputing resources to processes 560A-C.

Processes 560A-C may be instances of any computing services that canreceive, transmit, or process requests and perform one or more computingtasks in view of the requests. In one example, processes 560A-C mayinclude one or more web services, database services, filesystemservices, networking services, messaging services, load balancingservices, clustering services, configuration management services, otherservices, or a combination thereof. Each process may be executed oncomputer system 550 and may function as a client process, a serverprocess, or a combination thereof. The processes may execute on a singlevirtual machine or spread across multiple machines.

Predetermined resource 570 may be a software resource associated with aleader state of the cluster. Processes 560A-C may attempt to obtainexclusive access to predetermined resource 570 to enter the leader stateof the cluster. In an illustrative example, predetermined resource 570may be a file with a predetermined file name that may be created on apredetermined location in the file system. When one of processes 560A-Ccreates the file, the process enters a leader state and further attemptsto create the file by the other processes would fail. The file may havea predetermined name known to processes 560A-C as identifying the leaderstate marker file. Further, when the process in the leader state isterminated, an operating system process 555 may delete the file, thussignaling to the other processes that the leader state may be available.

Operating system process 555 may be responsible for monitoring thelifetime of the process in the leader state and releasing an exclusiveaccess to the predetermined resource 570 when the leader process isterminated. The operating system process may by any kernel process thatis responsible for cleaning up data items and files that are not in use.In an illustrative example, when the lifetime of the process in theleader state eventually ends, operating system process 555 may detectthat the leader process has been terminated and may delete the leaderstate marker file associated from the file system. In doing so, theoperating system process 555 signals processes 560A-C that the leaderstate may be available.

FIG. 6 is a flow diagram illustrating method 600 of monitoring thelifetime of a process in the leader state, according to one embodiment.At block 502 of method 600, process 1 of the plurality of processessuccessfully allocates ConfigMap object and enters into a leader state.In one example, Process 1 may need to enter the leader state to executea workload that may involve creating records in a data store thatenforces uniqueness constraints. When Process 1 enters the leader state,attempts to allocate the ConfigMap by the other processes in the clusterwould fail, thus preventing the other processes from entering the leaderstate.

At block 505, Process 1 may stay in the leader state through itslifetime. While in the leader state, an identifier of Process 1 isassociated with the ConfigMap as an owner reference. Process 1 maymaintain ownership of the ConfigMap through its lifetime and theConfigMap t may not be deallocated for the lifetime of the ownerprocess. Accordingly, other processes may not successfully allocate thesame ConfigMap.

Process 1 lifetime may eventually be terminated at block 510. Whenterminated, Process 1 no longer consumes memory space and associatedobjects and buffers may be deallocated from memory by a memorymanagement process. In one example, the memory management process may bea garbage collector process that may identify an object for deletionbecause the process identified by its owner reference is no longeralive. In one example, the garbage collector process may then deallocatethe object from memory. At 515, the garbage collector process mayrecognize that the ConfigMap has Process 1 identifier as the ownerreference. The garbage collector process may recognize that Process 1has been terminated because Process 1 may no longer have buffersallocated in memory.

At block 525, the garbage collector process may then deallocate theConfigMap from memory. When the ConfigMap is deallocated from memory, asecond process may be successful at allocating the ConfigMap. In orderto detect the availability of the leader state, the processes in thecluster periodically attempt to allocate the ConfigMap either atpredefined or calculated intervals of time. Thus at block 530, whenProcess 2 attempts to allocate the ConfigMap to enter the leader statein its next scheduled attempt, the attempt to allocate the ConfigMapwould be successful. Process 2 may allocate the ConfigMap and may enterthe leader state for the duration of its lifetime. The ConfigMap may nowhave Process 2 identifier assigned to its owner reference parameter.

FIG. 7 illustrates a diagrammatic representation of a machine in theexemplary form of a computer system 700 within which a set ofinstructions, for causing the machine to perform any one or more of themethodologies discussed herein, may be executed. The computer system 700may correspond to a node 100 of FIG. 1. In embodiments of the presentinvention, the machine may be connected (e.g., networked) to othermachines in a Local Area Network (LAN), an intranet, an extranet, or theInternet. The machine may operate in the capacity of a server or aclient machine in a client-server network environment, or as a peermachine in a peer-to-peer (or distributed) network environment. Themachine may be a personal computer (PC), a tablet PC, a set-top box(STB), a Personal Digital Assistant (PDA), a cellular telephone, a webappliance, a server, a network router, switch or bridge, or any machinecapable of executing a set of instructions (sequential or otherwise)that specify actions to be taken by that machine. Further, while only asingle machine is illustrated, the term “machine” shall also be taken toinclude any collection of machines (e.g., computers) that individuallyor jointly execute a set (or multiple sets) of instructions to performany one or more of the methodologies discussed herein.

The exemplary computer system 700 includes a processing device 702, amain memory 704 (e.g., read-only memory (ROM), flash memory, dynamicrandom access memory (DRAM) such as synchronous DRAM (SDRAM) or RambusDRAM (RDRAM), etc.), a static memory 706 (e.g., flash memory, staticrandom access memory (SRAM), etc.), and a secondary memory 716 (e.g., adata storage device), which communicate with each other via a bus 750.

The processing device 702 represents one or more general-purposeprocessors such as a microprocessor, central processing unit, or thelike. The term “processing device” is used herein to refer to anycombination of one or more integrated circuits and/or packages thatinclude one or more processors (e.g., one or more processor cores).Therefore, the term processing device encompasses a single core CPU, amulti-core CPU and a massively multi-core system that includes manyinterconnected integrated circuits, each of which may include multipleprocessor cores. The processing device 702 may therefore includemultiple processors. The processing device 702 may include a complexinstruction set computing (CISC) microprocessor, reduced instruction setcomputing (RISC) microprocessor, very long instruction word (VLIW)microprocessor, processor implementing other instruction sets, orprocessors implementing a combination of instruction sets. Theprocessing device 702 may also be one or more special-purpose processingdevices such as an application specific integrated circuit (ASIC), afield programmable gate array (FPGA), a digital signal processor (DSP),network processor, or the like.

The computer system 700 may further include a network interface device708. The computer system 700 also may include a video display unit 710(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), analphanumeric input device 712 (e.g., a keyboard), a cursor controldevice 714 (e.g., a mouse), and a signal generation device 716 (e.g., aspeaker).

The secondary memory 721 may include a machine-readable storage medium(or more specifically a computer-readable storage medium) 728 on whichis stored one or more sets of instructions 722 embodying any one or moreof the methodologies or functions described herein (e.g., leaderelection module 723). The instructions 722 may also reside, completelyor at least partially, within the main memory 704 and/or within theprocessing device 702 during execution thereof by the computer system700; the main memory 704 and the processing device 702 also constitutingmachine-readable storage media.

While the computer-readable storage medium 728 is shown in an exemplaryembodiment to be a single medium, the term “computer-readable storagemedium” should be taken to include a single medium or multiple media(e.g., a centralized or distributed database, and/or associated cachesand servers) that store the one or more sets of instructions. The term“computer-readable storage medium” shall also be taken to include anymedium other than a transitory medium such as a carrier wave that iscapable of storing or encoding a set of instructions for execution bythe machine that cause the machine to perform any one or more of themethodologies of the present invention. The term “computer-readablestorage medium” shall accordingly be taken to include, but not belimited to, solid-state memories, and optical and magnetic media.

The modules, components and other features described herein (for examplein relation to FIG. 1) can be implemented as discrete hardwarecomponents or integrated in the functionality of hardware componentssuch as ASICS, FPGAs, DSPs or similar devices. In addition, the modulescan be implemented as firmware or functional circuitry within hardwaredevices. Further, the modules can be implemented in any combination ofhardware devices and software components, or only in software.

Some portions of the detailed descriptions herein are presented in termsof algorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise, as apparent from the followingdiscussion, it is appreciated that throughout the description,discussions utilizing terms such as “receiving”, “generating”,“determining”, “using”, “assigning”, or the like, refer to the actionand processes of a computer system, or similar electronic computingdevice, that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

Embodiments of the present invention also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, or it may comprise a generalpurpose computer system selectively programmed by a computer programstored in the computer system. Such a computer program may be stored ina computer readable storage medium, such as, but not limited to, anytype of disk including floppy disks, optical disks, CD-ROMs, andmagnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic disk storage media, opticalstorage media, flash memory devices, other type of machine-accessiblestorage media, or any type of media suitable for storing electronicinstructions, each coupled to a computer system bus.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear as set forth in thedescription above. In addition, the present invention is not describedwith reference to any particular programming language. It will beappreciated that a variety of programming languages may be used toimplement the teachings of the invention as described herein.

It is to be understood that the above description is intended to beillustrative, and not restrictive. Many other embodiments will beapparent to those of skill in the art upon reading and understanding theabove description. Although the present invention has been describedwith reference to specific exemplary embodiments, it will be recognizedthat the invention is not limited to the embodiments described, but canbe practiced with modification and alteration within the spirit andscope of the appended claims. Accordingly, the specification anddrawings are to be regarded in an illustrative sense rather than arestrictive sense. The scope of the invention should, therefore, bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

What is claimed is:
 1. A method comprising: attempting, by a process ofa plurality of processes of a clusterized service, to obtain exclusiveaccess to a predetermined resource associated with a leader state of theplurality of processes; responsive to successfully obtaining theexclusive access to the predetermined resource, entering, by theprocess, the leader state; and staying, by the process, in the leaderstate for the lifetime of the process.
 2. The method of claim 1, whereinattempting to obtain exclusive access to the predetermined resourceassociated with the leader state further comprises: responsive tofailing to obtain the exclusive access to the predetermined resource,repeating an attempt to obtain the exclusive access to the predeterminedresource upon expiration of a timeout.
 3. The method of claim 1, whereasattempting to obtain exclusive access to the predetermined resourcecomprises attempting to allocate the predetermined resource.
 4. Themethod of claim 1 further comprises: responsive to determining that thelifetime of the process has ended, releasing, by an operating systemprocess, exclusive access to the predetermined resource.
 5. The methodof claim 4, whereas the operating system process is a garbage collectorprocess.
 6. The method of claim 1, wherein the predetermined resource isa ConfigMap object.
 7. The method of claim 1, whereas the predeterminedresource comprises an owner reference identifying the process in theleader state.
 8. A system comprising: a memory; and a processing deviceoperatively coupled to the memory, wherein the processing devicecomprises a plurality of processes of a clusterized service, and whereina process of the plurality of processes to: attempt to obtain exclusiveaccess to a predetermined resource associated with a leader state of theplurality of processes; responsive to successfully obtaining theexclusive access to the predetermined resource, enter the leader state;and stay in the leader state for the lifetime of the process.
 9. Thesystem of claim 8, wherein attempt to obtain exclusive access to thepredetermined resource associated with the leader state furthercomprises: responsive to failing to obtain the exclusive access to thepredetermined resource, repeat an attempt to obtain the exclusive accessto the predetermined resource upon expiration of a timeout.
 10. Thesystem of claim 8, whereas attempt to obtain exclusive access to thepredetermined resource comprises attempt to allocate the predeterminedresource.
 11. The system of claim 8 further comprises: responsive todetermining that the lifetime of the process has ended, release, by anoperating system process, exclusive access to the predeterminedresource.
 12. The system of claim 8, whereas the operating systemprocess is a garbage collector process.
 13. The system of claim 8,wherein the predetermined resource is a ConfigMap object.
 14. The systemof claim 8, whereas the predetermined resource is associated with anowner reference identifying the process in the leader state.
 15. Acomputer-readable non-transitory storage medium comprising executableinstructions that, when executed by a processing device, cause a processof a plurality of processes running on the processing device to: attemptto obtain exclusive access to a predetermined resource associated with aleader state of the plurality of processes of a clusterized service;responsive to successfully obtaining the exclusive access to thepredetermined resource, enter the leader state; and stay in the leaderstate for the lifetime of the process.
 16. The computer-readablenon-transitory storage medium of claim 15, wherein attempt to obtainexclusive access to the predetermined resource associated with theleader state further comprises: responsive to failing to obtain theexclusive access to the predetermined resource, repeat an attempt toobtain the exclusive access to the predetermined resource uponexpiration of a timeout.
 17. The computer-readable non-transitorystorage medium of claim 15, whereas attempt to obtain exclusive accessto the predetermined resource comprises attempt to create thepredetermined resource in a file system.
 18. The computer-readablenon-transitory storage medium of claim 15 further comprises: responsiveto determining that the lifetime of the process has ended, release, byan operating system process, exclusive access to the predeterminedresource.
 19. The computer-readable non-transitory storage medium ofclaim 15, wherein the predetermined resource is a file with apredetermined name.
 20. The computer-readable non-transitory storagemedium of claim 19, responsive to determining that the lifetime of theprocess has ended, deleting, by an operating system process, the filefrom the file system.